Fixed bug: Uncontrolled recursive calls that caused an infinite loop when loading certain pipelines containing Transformer2DModel #11923
Conversation
|
thanks @lengmo1996 |
Thanks for your quick reply @yiyixuxu . Here is a simple example. {
"_class_name": "Transformer2DModel",
"_diffusers_version": "0.34.0",
"activation_fn": "geglu-approximate",
"attention_bias": true,
"attention_head_dim": 88,
"cross_attention_dim": 512,
"dropout": 0.0,
"in_channels": null,
"norm_num_groups": 32,
"num_attention_heads": 16,
"num_embeds_ada_norm": 100,
"num_layers": 36,
"num_vector_embeds": 4097,
"sample_size": 32,
"norm_type": "ada_norm"
}config.json from diffusers import Transformer2DModel
transformer = Transformer2DModel.from_pretrained("./simple_demo")（I placed config.json in the simple_demo folder） |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
I pushed a commit to make the changed files conform to the ruff format requirements. It should pass the job of check_code_quality. |
|
I have checked the details of failing check about "LoRA tests with PEFT main". It seems that the failure is not related to this modification. This modification involves mapping Transformer2DModel to two variants, but pipeline_components is configured as UNet in the test that reports the error. I also tried to pull the main branch and test it locally with the help of the diffusers/diffusers-pytorch-cpu:latest image, and it seems that the code before the modification will also cause the test to fail. Due to my limited knowledge in code testing, I am not sure how to make further modifications so that this PR can pass all tests perfectly. What else can I do? |
…when loading certain pipelines containing Transformer2DModel (huggingface#11923) * fix a bug about loop call * fix a bug about loop call * ruff format --------- Co-authored-by: Álvaro Somoza <asomoza@users.noreply.github.com>
…when loading certain pipelines containing Transformer2DModel (huggingface#11923) * fix a bug about loop call * fix a bug about loop call * ruff format --------- Co-authored-by: Álvaro Somoza <asomoza@users.noreply.github.com>
What does this PR do?
Fix infinite recursive call issue when loading pre-trained weights of Transformer2DModel with some norm_type parameters
Fixes # (issue)
PR #7647 attempts to map Transformer2DModel to two different variants based on norm_type: {PixArt, DiT}-Transformer2DModel. However, some models may use other norm_type parameters, such as ada_norm or layer_norm. At this time, if you try to load a pre-trained model of a pipeline composed of models using such norm_type parameters, the program will enter an endless recursive call. Specifically, when the from_pretrained function of the pipeline is called, the from_pretrained function of Transformer2DModel will be called. Since Transformer2DModel inherits the LegacyModelMixin class, the from_pretrained function of the LegacyModelMixin class will be called. The from_pretrained function of the LegacyModelMixin class determines whether the Transformer2DModel needs to be mapped to a variant through the _fetch_remapped_cls_from_config function. When Transformer2DModel does not need to be mapped, it will fall into an endless recursive call: Transformer2DModel.from_pretrained → LegacyModelMixin.from_pretrained → _fetch_remapped_cls_from_config → Transformer2DModel.from_pretrained
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.